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Abstract 


Management  Science  Relations  for  Evaluation  and 
Management  Accountability 

Management  accountability  as^an  added  dimension  for  management  science 
research  is  examined  from  the  standpoint  of  possible  uses  in  some  of  the  newer 
"comprehensive  auditing"  approaches  to  propriety,  effectiveness  and  efficiency 
evaluations  of  management  and  organization  behavior.  Attention  is  centered  on 
non-market  activities  and  not-for-profit  organizations.  "Goal  focussing"  is 
examined,  for  example,  as  a  relatively  recent  extension  of  goal  programming  for 
use  in  effectiveness  evaluation  and  as  an  alternative  to  utility  theoretic 
approaches  in  national  goals  accounting  systems  designed  to  deal  with  programs 
or  objectives  involving  numerous  kinds  of  off-market  activities.  The  bulk  of 
the  paper,  however,  is  devoted  to  a  new  mathematical  programming  model  for  deriv¬ 
ing  analytic  representations  of  extremal  frontiers  or  envelopes  from  empirical 
data  and  for  measuring  the  efficiency  of  not-for-profit  entities.  An  illustra¬ 
tive  application  to  a  recently  completed  large-scale  social  experiment  in  edu¬ 
cating  disadvantaged  children  in  U.S.  public  schools  is  used  to  show  how  dis¬ 
tinctions  may  also  be  drawn  between  "program  efficiency"  and  "management  effi¬ 
ciency."  The  appendix  develops  a  canonical  form  for  the  types  of  statistical 
distributions  Involved.  It  also  provides  a  beginning  for  dealing  with  statistical 
properties  of  the  extremal  relations  obtained  by  applying  these  kinds  of  mathe¬ 
matical  programming  models  to  observational  data. 


1.  PERSPECTIVES 


The  occasion  of  a  plenary  address  can  be  used  in  many  ways.  On  this  occa¬ 
sion  we  want  to  try  to  provide  some  perspective  on  where  we  have  been  and  where 
we  can  (or  should)  be  going.  As  will  soon  become  clear,  we  shall  mainly  refer¬ 
ence  our  own  work,  but  we  shall  try  to  do  this  in  a  way  that  relates  this  work 
to  other  major  chrusts  in  the  development  of  management  sc  iec\ce  and/or  operations 
research.  Our  emphasis  will  be  directed,  for  the  most  part,  however,  towar  1  new 
paths  into  the  future  and  improved  ways  for  appraising  management  activities 
that  might  best  be  sumnarized  in  terms  such  as  "accountability"  and/or  "audit 
and  evaluation." 

In  broad  brush  fashion  the  past  history  of  management  science  may  be  idenfi- 
fied  with  first  an  evolution  from  mi Litary  to  private  sector  problems.  The  meth¬ 
odological  developments  accompanying  this  evolution  ensured  that  neither  the  mil¬ 
itary  nor  the  private  sector  applications  suffered  and,  indeed,  both  were  strength- 
ended.  This  continues  to  be  true,  we  think,  as  this  evolution  has  continued  into 
concern  with  now  public  sector  proble  ns  which  form  a  part  of  what  are  sometLoes  referred 
to  as  "public  management  science."  Tb a  point  to  stress,  we  believe,  is  that  all  three 
sectors — military,  private  sector  and  public  management  science  —  were  strengthened 
as  further  methodological  developments  occured  with  the  movement  into  this  new 
"public  management  science"  sector. 

Up  to  now,  the  emi  hasis  has  been  on  a  use  of  our  tools  and  concepts 
"in  the  service  of  mana;  ement,"  so  to  speak.1  Although  such  a  course  has  been 

1 fact,  under  the  leac ership  of  Gena  Woolsey,  this  has  become  a  guiding  principle 
in  the  editorial  policit s  of  Interfaces.  See,  e.g.,  R.  E.  D.  Woolsey  [22]. 

a  fruitful  one,  we  do  net  think  that  it  can  be  considered  as  controlling  in  all 
situations  and,  in  any  «ase,  we  do  not  think  it  represents  "the  end  of  the  line." 
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In  fact,  we  shall  argue  that  a  next  phase  in  the  growth  of  management  science 
can,  and  should,  be  directed  to  the  evaluation  and  control  of  management.  The 
trick  will  be  to  accomplish  this  in  a  way  that  continues  in  the  same  spirit  as 
before  —  viz. ,  effecting  these  ex  ensions  in  ways  that  will  strengthen  rathei 
than  weaken  what  has  already  been  achieved  by  management  science  "in  the  ser¬ 
vice  of  management"  (military,  private  enterprise,  and  public  service  varieties). 

Some  of  the  steps  that  have  already  been  taken  will  constitute  the  main 
focus  of  this  presentation.  How  this  further  evolution  in  our  discipline  can 
also  be  used  to  establish  contact  with  approaches  of  the  social  sciences  that 
have  also  been  concerned  with  such  evaluations  —  e.g.,  the  cost/benefit  stu¬ 
dies  of  economists  and/or  the  large-scale  social  experiments  designed  by  socio- 
gists  and  others  for  evaluating  programs  in  areas  like  education,  etc.,  —  will 
also  be  of  interest.  During  this  discussion,  however,  we  should  also  continue 
to  bear  in  mind  the  differences  in  emphasis  that  come  from  our  concern  with  man¬ 
agement  and  managed  entities. 

As  indicative  of  some  of  these  differences,  we  may  here,  for  the  moment, 
distinguish  between  "pure  predictions"  and  "control  predictions."  The  former 
may  be  secured, for  example,  from  commonly  employed  social  science  applications 
of  statistical  methods  to  ascertain  various  kinds  of  social  regularities  or  laws. 
The  latter,  l.e.,  the  control  predictions,  are  concerned  with  uses  of  statistics 
(and  like  analytical  devices)  to  ascertain  how  such  regularities  may  be  altered.* 


Charles  Christenson  has  called  our  attention  to  Karl  Popper's  distinction  be¬ 
tween  historical  phophecy  and  social  engineering  in  his  discussion  of  the  social 
science',  in  [  l.^),  where  similar  distinctions  are  made. 


in  a  similar  vein  we  may  also  distinguish  between  "research  for  discovery"  and 
"research  for  invention"  wherein  the  latter  is  more  closely  associated  with  the 
"control  prediction"  approach  that  we  have  just  described.  Note,  for  instance, 
that  the  usual  social  science  distinction  between  descriptive  (including  posi¬ 
tive)  and  normative  analysis  is  not  really  apt  here,  since  research  for  inven¬ 
tion  is  susceptible  to  tests  for  validity,  generalizability ,  etc.,  —  and  in  an 
objective  manner  —  just  as  is  the  case  in  "research  for  discovery."'*' 

^Without  attempting  to  push  the  matter  to  a  point  of  possible  conflict,  it  does 
seem  fair  to  say  that  it  will  usually  be  prudent  to  inquire  what  might  be  done 
to  change  the  phenomena  being  studied,  as  is  always  necessary  in  "research  for 
invention,"  even  when  "research  far  discovery"  and/or  "pure  prediction"  are  pri¬ 
mary  research  objectives.  See,  e.g.,  Charles  Ferrow  [ 17 ] . 

There  are,  however,  differences  that  should  be  noted.  For  instance, 'in  research  for  in¬ 
vention  we  may  want  to  focus  our  analyses  on  only  subsets  of  the  available  data  instead 
of  simply  examining  all  of  the  data  for  their  possible  value  as  in  discovery  re¬ 
search.  The  results  from  the  analyses  of  these  subsets  for  invention  research 
may  then  perhaps  be  applied  to  other  data,  as  we  shall  shortly  illustrate,  in 
an  attempt  to  see  how  operations  of  an  entire  group  of  entities  may  be  altered 
or  improved.  As  a  case  in  point  we  might,  for  instance,  study  subsets  of  data 
for  entities  that  have  improved  their  energy  consumption  efficiencies.  This 
might  be  done  to  enable  us  to  ascertain  how  to  improve  the  operations  of  other 
entities  or  else  enable  us  to  estimate  wastes  and  inefficiencies  associated  with 
not  doing  this.  Such  an  approach  would  require  recourse  to  models  and  methods 
that  will  differ  in  important  respects  from  those  that  are  currently  being  emplov  • 

in  econometric  studies  in  U.S.  energy  problems  and  the  same  is  ur^ 
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of  social  science  modeling  in  other  areas  as  well. 

Further  discussion  may  be  found  in  Charnes,  Cooper  and  Schinnar  [9  ]. 
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2.  SELECTION  OF  TOPICS  AND  RELATIONS  TO 
COMPREHENSIVE  AUDIT 


We  have  now  outlined  the  course  we  propose  to  follow,  including  the  con¬ 
tacts  that  might  be  made  with  other  ;cien  es  —  along  with  some  of  the  kinds  of 
methodological  and  conceptual  altera  lions  that  this  might  require.  We  will  also 
do  well  to  indicate  aspects  of  managerial  practice  that  may  be  served  by  '■he 
kinds  of  developments  we  are  conside  "in&  and  for  this  we  turn  to  "audit  practices" 
and  processes — because  of  the  "3rd  party  accountability  functions"^  that  these 

^A  discussion  of  these  3rd  party  accountability  functions  and  how  these  relate  to 
accounting  and  auditing  may  be  found  in  [ 13-2 ]  and  [13.3]. 

processes  are  supposed  to  serve. 

The  term  audit  is  sometimes  viewed  as  synonymous  with  the  CPA  function  of  at¬ 
testing  to  financial  statements  and  related  representations  by  management.  This 
CPA  kind  of  audit  has  the  servicing  jf  "3rd  parties"  as  a  major  function  and  it 
therefore  carries  a  corollary  responsibility  for  objectivity  and  professional 
canons  of  care  and  validation  that  are  of  interest  to  us.  Such  "financial  audits" 
form  only  a  part  of  the  kinds  of  audits  that  are  now  being  practiced,  however,  and 

among  the  class  of  such  wider  possibilities  we  shall  single  out  those  that  we  have 

2 

elsewhere  referred  to  as  "comprehensive  audits." 


2See  [12]. 


The  term  "comprehensive"  seems  appropriate  because  this  type  of  audit  extends 
the  process  of  objective  appraisal  t>  all  aspects  of  management.  By  this  we  mean 
that  the  auditor  assumes  responsibility  for  the  aspect  of  management  that  is 
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designated  for  audit  as  well  as  the  vay  the  audit  is  conducted.  The  auditor  d'  e- 
not  stop  short  with  the  attest  function  under  comprehensive  audit.  Instead, he 
also  assumes  full  responsibility  for  the  report  that  is  render. -d ,  as  well  as  the 
3rd  party  groups  to  be  serviced  by  the  audit  process.  Finally,  instead  of  restrict 
ing  audit  examinations  to  financial  transactions .and  their  representations  in  fin¬ 
ancial  reports)  such  comprehensive  audits  may  extend  to  examination  and  appraisal 
of  any  of  the  following  aspects  of  n  anagement  and  organization  behavior*. 

1.  Propriety  of 

a.  Objectives  pursued 

b.  Methods  used 
1.  Effectiveness  in 

a.  Stating  objectives 

b.  Attaining  objectives 

3.  Efficiency  of  performa  ice  as  measured  by 

a.  Benefits  received 

b.  Resources  utilized. 

For  the  sake  of  concreteness,  we  should  observe  that  such  audits  are  not  merely  constructs 
from  our  imaginings.  They  are  presently  employed  by  the  U.S.  General  Accounting  Office  and 
they  may  also  be  found  in  the  internal  audit  practices  of  certain  large 
multinational  enterprises.  Moreover  the  value  of  these  kinds  of  audits  are 
now  being  recognized  by  other  governmental  units,  both  foreign  and  domestic, 
and  they  are  likely  to  continue  to  g  ow  as  part  of  the  internal  control  and 

corporate  governance  reforms  that  ar<  presently  being  considered  by  many 

2 

private  enterprises.  In  any  case,  w<  shall  use  the  possible  needs  and 

^See  Churchill,  et  al.  in  [12 ] - 
2 

Including  the  extension  of  the  audii  function  that  is  represented  by  the  Peer  Re¬ 
view  audits  of  CPA  firms  themselves  i  hat  are  discussed  in  119], 


6 


opportunities  of  such  comprehensive  audits  to  guide  our  discussion  since  via  its 

3rd  par  tv  or ientat Lon,  this  kinu  of  a  idit  also  points  in  thu  new  directions  that 
we  want  to  explore  for  management  science. 

In  this  paper  our  focus  will  be  on  recently  developed  methods  for 
objectively  assessing  various  kinds  of  ef f iciency^  that  are  pertinent  to 

— ^We  use  this  term  ir  the  sense  indicated  ibove  (where  it  is  distinguished 
from  effectiveness  and  propriety). 

the  activi; ies  of  not-for-profit  entities.  Hence  we  first  ought  to  say  a  word 
about  the  issues  of  propriety  and  effect  iveness,  which  form  highly  important  parts  of  a 
widened  maragement  accountability  that  is  now  beginning  to  be  recognized.  On  the  subject  of 
"propriety',  we  should  probably  note  that  there  is  a  tendency  on  the  part  of  auditors  to 

use  this  term  to  cover  situations  which  range  from  criminal  activities  of 
the  grossest  sort  down  to  trivial  discrepancies  in  records  or  the  way  they 
have  been  maintained.  This  tends  to  underemphasize  the  importance  of  the 
former  and  to  overemphasize  the  importance  of  the  latter.  Somewhere  within 
these  two  extremes,  moreover,  there  is  a  need  for  perspective  that  is  also 
often  missing.  After  all,  some  of  the  improprieties  of  yesterday  have  become 
the  propr.^f'es  of  today  and  the  former  my  even  have  been  the  precondition 
for  progress  from  one  to  the  other.  How  and  in  what  form  such  improprieties 
may  be  ingredients  of  social  progress  is  a  subject  that  is  still  only  poorly 
understood  and  hence  we  need  to  approach  this  in-between  area  with  same 
tolerance  and  perspective. 

Having  said  the  above,  however,  one  must  allow  for  the  fact  that 
until  "propriety"  can  be  assumed,  it  is  extremely  difficult  to  bring  the  issues 
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of  effectiveness  or  efficiency  Into  perspective.  Witness,  for  Instance,  the 
fact  that  in  thousands  of  pages  of  testimony  dealing  with  the  conduct  of  activities 
in  sensitive  Federal  government  agencies  1  ike  the  CIA  and  FBI  there  is  scarce  1%  any 
discussion  of  effectiveness  or  efficiency.  These,  more  positive, aspects  of 
agency  operations  must  (or  should)  le  brough,  into  prominence  sooner  or  later, 
however,  and  an  unmet  challenge  of  fur  time  is  how  to  control  and  improve  the 
operations  of  these  kinds  of  agencies. 

''Further  discussion  and  references  nay  be  found  in  Churchill  and  Cooper  [1]  ]. 


The  difficult  part  of  the  luestlon  of  effectiveness  lies,  of  course, 
in  the  choice  (and  statement)  of  objectives.  This  has  been  a  prime  concern 
of  research  in  the  area  of  decision  theory  including  the  more  recent  multi¬ 
attribute  extensions  for  use  on  the  multiple  objective  decision  problems 

2/ 

that  normally  confront  not-for-profit  enterprises.-  Something  more  seems 

2/  ~ 

—  See  Keeney  and  Raiffa  [  14l  ]. 

to  be  needed,  however,  in  moving  fron  the  sphere  of  individual  choice  to  public 
accountability  including  due  allowance  for  degrees  of  possible  inconsistency  and 
che  adaptation:  that  are  likely  to  be  required  in  the  continually  changing  sets  of 
social  values  where  many  not-f  jr-profit  entities  now  operate. 

Experiments  in  mixing  a  variety  of  approaches  would  seem  to  be  in 
order.  As  par;  of  a  consortium  concerned  with  developing  a  national  goals 
accounting  system,  for  instance,  we  f re  utilizing  an  approach  that  we  refer 
to  as  "goal  focusing."—^  In  this  aj  proach  the  elements  of  "goal  programming”^ 

3/  — 

-  See  the  discussion  of  "goal  focusii  g"  in  [10] 

— ^ See  Charnes  and  Cooper  [  3]. 
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are  combired  with  "efficient  point"  onsiderat ions  so  that  a  choice  is  effected 
from  amonj  the  latter  set.  In  this  /ay  the  goal  programming  approach  focuses 
on  only  a  subset  of  the  entire  set  o  efficient  points.  Note,  however,  that  the 
closest  efficient  point  may  be  associated  with  solutions  that  are  further  from 
the  goals  than  would  be  the  case  if  .>ther  (non-efficient)  points  wert  lesignated. 

In  the  development  of  these  "goal  focus"  approaches  we  are  trying  to  con¬ 
nect  so-called  "qual ity-of-life"  measurements  with  their  economic  costs  and  con¬ 
sequences  as  part  of  a  national  goals  accounting  system.  Hence  the  needs  and  ap¬ 
proaches  utilized  by  economists  need  to  be  kept  in  mind.  The  characteristic  ap¬ 
proach  in  formal  economics  is  one  which  proceeds  by  maximizing  a  "utility  func¬ 
tion"  under  a "budget  constraint."  Ar  advantage  of  this  approach  is  that  it  re¬ 
stricts  attention  to  a  relatively  snu  11  subset  of  the  entire  set  of  efficient- 
point  possibilities  while,  at  the  sane  time,  allowing  a  study  of  tradeoff  possi¬ 
bilities  by  reference  to  pricing  alternatives.  The  goal  focus  approach  also  re¬ 
stricts  attention  to  a  subset  of  the  efficient  points  but  does  this  in  a  way  that 
relates  tie  resulting  solution  to  the  goal  sets  while  also  yielding  the  tradeoffs 
in  terms  of  dual  evaluators  associat ad  with  deviations  from  these  goals. 

The  latter,  i.e.,  the  goal  sets  and  their  relations  to  each  other,  are  ob¬ 
tained  i-  •  arious  ways  which  include,  e.g.,  the  responses  that  one  might  get  from 
interview  or  questionnaire  data  such  as  might  be  secured  from  household  panels. 

Many  of  the  goals  that  are  pertinent  to  quality-of-life  dimensions  relate  to  off- 
market  considerations.  Hence  it  seems  unvise  to  retain  market  price  related  be¬ 
havior  as  the  center  of  all  of  the  analysts.  The  goal  focus  approach  relaxes  this 
requirement,  but  in  a  way  that  maintains  •  ontact  with  the  efficient  point  considera¬ 
tions  of  economic  analysis  so  that,  inter  alia,  sensitivity  and  trade-off  analyses  can 

^See  the  discussion  in  Chapter  IX  of  (  i  ]. 
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be  conducted  in  terms  of  the  "real”  'cor.omic  and  other  costs  which  are  pertinent 

for  public  policy  applications.  The  point  is  that  these  costs  and  tradeoffs  are 

more  fundamental  than  any  utility  theoretic  formulation  and  all  of  them  need  to 

be  accommodated  in  one  systematic  an,  1  .sis  such  as  the  goal  focus  approach  is  de- 

signed  to  provide  via  (a)  a  focus  on  only  a  subset  of  the  efficient  points  and  (b)  summary  mea- 

u:  ••••:  ->f  t!oe  tradeoff  possibilities  via  t  i"  optimal  dual  variables  associated  with  devi.i- 

1  / 

t  '  n  ■  r  r;r,  the  goal . 

^These  topics  and  others  are  treated  in  detail  in  a  forthcoming  consortium  re¬ 
search  volume  entitled  The  Product  lor  of  Well  Being  which  will  be  published  by 
the  National  Planning  Association  un< er  the  editorship  of  N.  Terleckvj.  See  [10] 

See  also  Schinnar  [21]. _ _ _ _ _ 

There  are,  of  course,  many  questions  that  remain  and  these  include  the  kinds 
of  efficiency  estimates  and  ad;ustments  that  might  be  made  and  how  these  might  be 
objectively  validated.  Other  aspects  of  the  topic  of  efficiency  will  occu¬ 
py  us  in  the  remainder  of  this  paper,  which  will  be  developed  as  follows.  First, 
we  snail  indicate  what  we  mean  by  efficiency  and  how  we  propose  to  measure  it.  Then 
we  shall  apply  it  to  data  secured  frcm  Program  Follow  Through,  a  recently  completed 
large-scale  social  experiment  In  U.S  public  school  education.  This  will  provide  an 

opportunity  to  distinguish  further  between  what  we  shall  refer  to  as  "progi am 
efficiency"  and  "management  efficiency','  in  which  we  may  visualize  the  latter 
as  representing  the  extent  to  which  nanagement  takes  advantage  of  the 
opportunities  provided  by  the  former.  Finally,  we  shall  attempt  to  provice 
a  summing  up  and  we  shall  also  proviie  an  appendix  that  constitutes  a  stai  t 
toward  the  kinds  of  further  statist! :al  developments  that  are  needed  in  dealing 
with  tl ese  kinds  of  issues. 
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A  Model  for  Kfficiencv 


„  1/ 

Measurement- 


To  initiate  cur  discussion  in  this  section  we  may  observe  that  it  is  customary 
to  measure  efficiency  as  a  ra  :io  of  want  d  outputs  to  valued  inputs  both  in  non-nega¬ 
tive  measure.  The  value  of  this  ratio  is  at  most  unity  and  hence  we  may  write 
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to  represent  the  kinds  of  ratios  that  wt  will  be  considering  to  measure  the  efficiency  of 
each  of  j  =  1,  ....  n  decision  making  units  (DMUs).  Here  the  DMUs  will  be  associat¬ 


ed  with  sites  in  various  school  districts  with  the  y  .  and  x  (all  positive)  re- 


p: esenting,  respectively,  agreed  upon  outputs  and  inputs  such  as  we  shall  shortly  be 

2/ 

discussing.— 


Because  the  observed  y^  and  ai  e  all  assumed  to  be  positive  and  because  the 
weights  uf,  v^  are  also  be  restricted  tc  positive  choices,  the  non-negativity  imposed 
on  the  ratio  in  (1)  will  always  be  satisfied.  Hence  it  will  be  eliminated  from  fur¬ 


ther  explicit  attention.  The  problem  ol  selcting  the  uf,v^  for  rating  any  DMU  in  an 
objective  manner  is  then  resolved  by  introducing  an  optimizing  principle  as  follows: 


*;his  section  draws  heavily  on  research  done  in  collaboration  with  E.  Rhodes  as  re¬ 
ported  in  [7]  and  f 2<J J  - 

2 

Other  entities  might  also  be  used,  of  course,  including  internal  organization  units, 
etc.  See  pp.  430-431  in  [7]. 
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The  resulting  values  of  the  u^,  are.  then  determined  entirely  from  the  , 

v  data  in  terms  of  this  model.  Here  the  y  and  x.  represent  the  data  for 
' r j  ro  10 

one  of  the  j  =  1,  n  DMUs  that  has  been  singled  out  for  efficiency 

evaluation  so  that  evidently  the  rating  is  to  be  effected  relative  to  the 
other  DMUs  in  the  set.  That  is,  the  result  is  a  rating  of  relative  efficiency. 

Finally,  because  the  ratio  in  the  functional  also  appears  in  the  constrain' s 

*  * 

we  have  max  h  =  h  <1  and  h  =1  only  if  DMU  is  efficient, 
o  o  o  ° 

As  formulated  in  (2)  we  appear  to  be  required  to  deal  with 
optimization  of  a  linear  fractional  functional  under  linear  fractional 
constraints.  Because  we  want  to  foc» s  on  conceptual  clarity,  we  will  not 
pursue  this  topic  in  detail  but  we  uuy  say  that  this  problem  can  be  replaced 
by  an  ordinary  linear  programming  equivalent Moreover,  this  linear 


— ^See  [  7  ]. 
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programming  problem  and  its  dual  provide  not  only  the  wanted  measures  of 
efficiency  but  also  the  marginal  productivities  (from  the  efficient  DMUs)  and 
these  values  can  be  used  to  construct  the  efficient  production  possibility 
surface.  Thus,  in  addition  to  obtair ing  the  wanted  efficiency  meisures,  a 
new  simultaneous  estimation  method  fc r  obtaining  extremal  relations  from 
empirical  data  is  also  provided. 

The  immeciately  preceding  remarks  will  not  be  pursued  further  in 
this  paper  because  we  want  to  center  all  of  our  remaining  attention  on  these 
efficiency  measures.  They  are  introduced  here  only  to  help  us  interpret  the 
results  of  the  optimization  in  (2).  This  we  may  do  with  reference  to  the  illus¬ 
trative  data  of  Table  1,  which  is  supposed  to  represent  a  situation  in  which  3 
DMUs  produce  the  same  single  unit  of  output.  The  data  in  the  rows  for  x.^  and  x2 
represent  the  amounts  of  the  inputs  utilized  by  DMU^,  DMU2  and  DMU.J,  respectively, 
in  the  production  of  this  same  one  unit  of  output.^- 


^We  may  also  think  of  this  situation  as  the  input  amounts  per  unit  output  of 
these  3  DMUs. 


Evidently,  DMU2  is  not  as  fficient  as  DMU^  and  hence  cannot  be 

* 

characterized  as  efficient  (i.e.,  it  cannot  have  h  =  1)  since  DMU  has 

o  x 

produced  the  same  amount  of  output  ( me  unit)  with  one  unit  less  of  the  first 

2 

input  (i.e.,  we  have  x^  *  2  vs.  x^2  *  3)  and  no  more  of  the  second  input. 


Formal  definitions  of  the  efficiency  concepts  being  used  here  are  provided  in  [5]  . 
and  [20]  where  they  are  also  related  to  other  concepts  such  as  Pareto  efficiency  ^ 
in  economics .  * 


t 
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TABLE  1 

An  Illustration  of  DM1 J  (=  Managerial/ Efficiency 


-  14 


the  input  amounts  utilized  by  DMUj. 
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Figure  1  portrays  the  situation  geometrically.  Here  P 2  corresponds 
to  the  data  for  DMU^  and  and  P^  similarly  correspond  to  the  data  for  DMU^ 

and  DMU  ^  when  the  axes  are  interpreted  as  providing  coordinate  values  for  the 

* 

input  variables  recorded  in  Table  1.  The  contraction  associated  with  hQ  =  6/7 
brings  ?2  into  coincidence  with  P2  where  the  latter  lies  on  the  "unit  isoquant" 
—  i.e.,  the  isoquant  segment  represented  by 

(A . 1)  {(x1>x2)  :  |  ^  +  f  x2  =  1;  2  <  xL  i  A;  1  <  *2  <  2}  . 

Turning  to  the  situation  depicted  for  P^  in  Figure  1,  a  similar 

■k 

analysis  would  also  give  hQ  =  6/7  but  now  the  relevant  isoquant  segment  would  be 


(A. 2)  {(xt,x9)  :  |  Xx  +  \  x2  =  1;  1  -  -  2!  2  ~  *2  -  41 


Thus,  although  the  contraction  factor  is  the  same  in  both  cases,  the  efficient 
referents  differ  —  since  the  pertinent  isoquant  segments  are  generated  from  P^ 

and  P,  in  the  case  of  P.  and  fr<m  P,  and  P-  in  the  case  of  P„. 

1  A  1  i  2 

In  a  loose  sense  this  difference  in  the  referents  provides  some  advantage 
for  the  applications  we  shall  be  studying  insofar  as  P,.  and  P^  generate  a  cone 
which  is  more  pertinent  for  P^  than  the  cone  generated  from  P^andP^.  See  f  5], 
Instead  of  pursuing  the  sense  of  the  advantages  from  this  choice  of  referents, 
however,  we  prefer  to  continue  with  our  discussion  of  Figure  1  in  terms  of  the 
results  portrayed  in  (A.l)  and  (A. 2).  The  values  v^  =  1/6  and  =  1/3  in  (4.1) 
represent  the  marginal  productivities  obtained  from  the  efficient  isoquant  seg¬ 
ment  connecting  P^  and  P^.  These,  as  already  observed, are  the  efficient  produc¬ 
tivities  obtained  from  the  data  for  and  P^.  When  applied  to  the  inputs  util¬ 
ized  by  P2  =*  these  productivity  s  give 
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(5.1)  3v*  +  2v*  =  7/6. 

Thus  we  may  think  of  this  application  of  (2)  to  the  data  of  Table  1  (or  Figure  1) 
as  positioning  P ^  on  the  isoquant  segment  for  the  output  level  7/6  —  which 
would  lie  to  the  northeast  and  parallel  to  the  segment  of  the  unit  isoquant  con- 

i 

necting  and  P^  in  Figure  1. 

In  a  similar  way  we  could  also  apply  the  slope  values  shown  in  (4.2)  to  the 

data  for  P.  to  obtain 
4 


so  that  this  same  isoquant  level  would  again  be  attained,  but  now  it  would  be 
parallel  to  the  segment  between  P^  and  P^. 

The  interpretation  is  now  staight forward  in  either  case.  Our  model  (2) 
obtains  the  productivities  from  the  e  Tf icient  DMUs  and  applies  them  to  all 
other  DMUs  to  estimate  what  ouputs  could  have  been  obtained  from  the  inputs 
utilized  by  the  latter  in  each  case.  In  other  words,  the  model  (2)  applied  to 
the  data  shown  in  Figure  1  has  given  us  the  ability  to  construct  any  part  of 
the  entire  production  function  surface  which  may  be  pertinent.  See  [  5], 

Up  to  this  point  we  have  restricted  attention  to  the  single  output  case  so 
that  we  could  thereby  have  easy  access  to  the  customary  production  function  and 
related  concepts  from  elementary  ecoromics.  The  numerator  of  (2)  involves  mul¬ 
tiple  outputs,  however,  and  so  the  concept  of  a  production  function  must  give  way 
to  more  general  concepts  for  a  full  interpretation  of  these  ratios.— ^  We  want  to 

-  See  the  discussion  of  production  possibility  surfaces  in  [7] 
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effect  our  interpretation  in  a  way  that  maintains  contacts  with  engineering  as  well 
a;  economics  constructs^  however,  and  this  dictates  the  way  we  will  now  proceed. 

It  is  customary  to  measure  efficiency  in  engineering  when  there  is  a  single  po¬ 
sitive  input  and  a  single  positive  output  by  the  ratio  of  the  output  to  the  input, 
bcth  measured  in  the  same  dimensional  units  (e.g.,  in  energy  units).  An  energy  in¬ 
put  is  transformec  by  the  process  being  examined  into  an  output  energy  in  a  diffe¬ 
rent  form.  By  the  second  law  of  thermodynamics  such  a  transformation  cannot  in¬ 
crease  the  amount  of  energy  in  the  new  form  over  that  in  the  original  (input)  form. 

We  generalize  this  idea  to  other  contexts,  or  "production  processes,"  by  in¬ 
troducing  a  single  "virtual  input"  and  i  single  "virtual  output"  obtained  by  multi¬ 
plying  each  nput  by  a  weight  and  summirg,  an  l  by  also  multiplying  each  output  by 
a  weight  and  summing  as  was  done  in  (2).  We  do  not  restrict  the  dimension  of  the 
virtual  input  and  output  other  than  to  require  that  they  be  the  same  in  order  to 
yield  a  dimensionless  ratio.  Further,  we  require  of  this  dimension  for  the  virtual 
input  and  virtual  output  that  it  have  the  property  of  energy  conversion,  viz ,  that 
the  virtual  output  cannot  exceed  the  virtual  input. 

To  rate  the  efficiency  of  a  particular  DMU  —  say  DMUq  —  we  formulate  the  ex¬ 
tremal  principle  of  choosing  the  weights  so  as  to  satisfy  the  "energy-degradation" 
principle  for  all  the  DMU’s  such  that  the  virtual  output  to  virtual  input  ratio  of 
DMUq  is  a  maximum.  This  maximum  value  represents  its  efficiency. 

All  of  this  is  Hone  analytically  in  (2).  It  should  also  be  explicitly  noted 
that  our  efficiency  rating,  as  obtained  in  (2),  is  independent  of  the  units  in  which 

7  * 

Further  extension  in  the  economics  direction  would  identify  the  ur  in  (2)  with  effi¬ 
cient  "margiral  rates  of  transformation"  in  a  manner  parallel  to  the  identification  of 
the  v*  with  the  marginal  productivity  constructs  that  we  have  already  effected  from 
economics.  See  Allen  [1]. 
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Che  observed  inputs  and  outputs  are  measured  so  long  as  the  units  are  the  same 

for  everv  DMU.  To  see  that  this  is  so  we  need  only  note  that  if  the  x..  for  a 

t.J 

particular  DMU  are  replaced  by  x ^  =  ox^.  >  0  for  j  =  1,  ....  n  then  replacing 

★  * 

v^  (an  optimal  value  of  v^)  by  v!^  =  v_.  fo  and  leaving  all  other  u's  and  v's  un- 

* 

changed  we  satisfv  all  contraints  and  obtain  hf  =  n  .  Thus  we  must  have  maic  hf  = 

°  o  o  o 

,  ^  ^  •jr 

h'  >  h  ,  On  the  other  hand,  if  we  could  have  h'  >  h  ,  then,  bv  the  inverse 
o  —  o  o  o 

*  * 

transformation,  we  could  obtain  an  h  from  u'  ,  v'  vhict,  would  satisfy  the  ori- 

o  J 

einal  constraints  and  we  would  also  have  h„  =  h'*  >  h*.  But  this  contradicts  the 

°  o  o 

assumed  maximality  cf  h  ,  and  so  we  must  have  h'  =  h  . 

o  oo 

Thus  we  obtain  the  same  efficiency  value  regardless  of  the  units  in  which 
the  itfl  input  is  measured.  A  similar  argument  holds  for  any  other  input  or  out¬ 
put.  Our  efficiency  measure  is  therefore  independent  of  the  units  employed  in 
any  of  the  inputs  or  outputs  and  hence  these  units  may  be  chosen  for  computa¬ 
tional  (or  other)  convenience  in  any  fashion. 

The  background  we  have  now  supplied  provides  access  to  the  well-known  com¬ 
putational  power  and  computer  code  availablities  of  ordinary  linear  programming. 
This  means  that  large  numbers  of  inputs,  outputs  and  DMUs  can  be  handled  without 
any  great  difficulty.  The  need  for  this  kind  of  power  and  convenience  is  becom¬ 
ing  increasingly  apparent  in  energy  (and  other)  studies  which  can  no  longer  be 
adequately  addressed  by  only  econometric-statistical  techniques  and  models.— ^ 

The  interpretations  we  have  supplied  made  direct  contact  with  both  engineering 

2/ 

and  economics  definitions  of  efficiency;-  One  can,  of  course,  continue  to  use  these 


See,  e.g.,  Manne,  et^  al.  [16], 


1/ 

2/ 

—The  u*,  v*  values  have  economic  significance  in  their  own  right  as  efficient  mar¬ 
ginal  productivities  and  transformation  rates.  To  emphasize  their  possible  use  sepa¬ 
rate  from  the  above  models  and  developments  we  refer  to  them  as  components  oi  aPrcduc- 
tive  Efficiency  Vector  represented  by 


PEV* 


. u*;  v*,  vj . v*). 
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in  manners  Chat  have  been  customary,  although  even  then  new  possibilities  for  ap¬ 
plication  are  thereby  opened.  For  instance,  one  might  proceed  by  other  means  to 
estimate  the  efficient  productivities  as  a  basis  for  determining  savings  in  areas 
like  energy  consumption  by  altering  D'iU  practices  for  this  and  other  outputs  of 
their  operations  that  might  be  of  interest  instead  of  proceeding,  as  at  present, 
merely  to  estimate  projected  patterns  of  energy  consumption  from  past  practices 
and  efficiency  distributions,  as  is  tie  practice  in  presently  conducted  economet¬ 
ric  studies Finally,  our  approach  opens  still  further  possibilities  for  new 
and  important  distinctions  such  as  we  shall  next  undertake  to  illustrate  by  distin 
guishing  between  "program  efficiency"  and  "management  efficiency"  for  their  poten¬ 
tial  importance  for  various  kinds  of  evaluations  and  accountability  relations. 


—See,  for  instance,  the  Kennedy  and  other  models  discussed  in  [16].  See  also 
[9]. 
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l_f  •  .JiliXxS  u. ri-w-n.  fulAjPftL.N  1  AJ<  Ajj  1  »>  X  — 

Figure  2,  below,  will  help  as  to  introduce  our  next  topic. 

'*•;  may  think  oi  this  ns  a  portrayal  in  which  inputs  in  amounts  x  tur/e 
resulted  in  observed  counts  y  of  a  s  .ngie  output  (all  else  held 
constant J  for  a  collection  of  ufflJ's.  ihcre  is  a  further  distinction 
in  that  these  bird's  operate  ’under  2  d.  fercnt  technologies  so  that 
the  efficient  frontiers  for  one  set  a  :e  at  3  while  the  efficient 
frontiers  of  the  other  set  are  at  .evidently  even  efficient  LMJ's 
in  3  cannot  achieve  the  levels  of  tho  is  in  A  and  aver  some  cf  the  lest 
efficient  nMO's  operating  under  u  exc  ied  what  is  attainable  under  3. 

A  standard  approach  via  sta  istical  regressions  might  have  concealed 
some  of  the  alternatives  for  choice  that  are  present  because  the  ebservatiens 
generally  contain  a  mix  of  technological  possibilities  with  their  utilisa¬ 
tion.  ihis  is  the  situation  we  would  now  like  to  confront  by  extending 
our  previous  analysis  in  order  to  breng  it  to  bear  in  . ays  that  might  help 
us  distinguish  between  what  we  shall  refer  to  as  "managerial  efficiency" 
and  "program  efficiency",  The  latter  may  be  thought  af  in  terns  of  the  kinds 
oi  frontiers  depicted  in  Figure  2  while  the  former  may  contain  inefficien¬ 
cies  resulting  from  managerial  decis:ons  that  fail  to  utilize  these  oppor¬ 
tunities  to  the  full. 

"Data  envelopment  /.na^ysis"  is  the  name  that  we  shall  use  for 
the  approach  we  are  suggesting  1 ccaut  i  (a)  it  first  tries  to  locate  the  boun¬ 
daries  that  envelop  the  observ  .tions  as  in  A  and  B  of  Figure  2  and  then  (b)  it 
brings  the  observations  all  up  to  the  envelope  that  is  pertinent  in  each 
case.  Finally,  it  imputes  any  remaining  efficiency  difference  to  the 
respective  programs  so  that  in  the  sc  tuation  of  Figure  2,  say,  the  program 

1/  This  part  of  our  paper  draws  heaviiy  on  work  done  in  collaboration  with 
E.  Rhodes  as  reported  in  [6].  See  also  [20], 
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aocociatad  with  i.  would  be  characterized  aj  more  -.If  icient  than  the 
program  associated  with  B.  Moreover,  the  numerical  value  cf  these 
efficiencies — measured  as  the  differ*  nee  in  the  statistical  distribu¬ 
tions  resulting  from  the  indicated  a*  jus  talents — is  intended  to  repre¬ 
sent  the  mount  of  output  gain  that  s  attainable  by  moving  the  uMLT's 
from  B  to  A. 

The  above  remarks  need  further  clarification  which  we  may 
obtain  by  our  earlier  contrast  between  "control  predictions"  and  "pure  predic¬ 
tions."  The  latter  term,  we  may  recall,  is  intended  to  refer  to  the  kind 
of  predictions  that  one  might  make  from  the  customary  statistical 
approcor.es,  fitting  regression  e qua  .ions  to  aid  B  separately, 
without  otherwise  distinguishing  between  the  observations, 

for  instance,  might  lead  to  results  .n  which  the  program  associated 
with  B  was  evaluated  as  being  the  be  .ter  of  tno  tw'o  alternative  -:. 

This  pure  prediction  approach  could  be  justified  if  it  were 
assumed  that  ail  DMU's  would  bo  perm  .tied  to  continue  .o  operate  at 
the  seme  levels  of  managerial  effici mcy.  If,  however,  ne*  controls 
to  alter  this  past  pattern  of  behavior  could  be  considered,  then  an  alter¬ 
nate.  approach  might  be  used  in  the  form  of  a  "control  prediction"  co  Justi¬ 
fy  the  choice  of  A  over  B. 

D_uv,  Data  envelopment  Analysis,  is  intended  for  ise  in  the 
latter  situations  since,  without  suci  controls,  the  result:  ng 
"elficient  behavior"  predictions  wil  generally  be  invalid  V.e  may, 
in  fact,  think  of  Dua  as  providing  guidance  for  program  auuits  which  will 
help  to  locate  sources  of  inefficient  in  order  to  direct  such  audit  ex- 

1  t 

aminations  to  particular  DMU's  brougl  t  into  view  by  a  DEA  analysis.—'  A 
separate  decision  may  then  be  made  concerning  whether  the  projected 

^Further  discussion  is  provided  in  [2], 
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efficiencies  can  be  realized  and  perhaps  thereby  justify  a  choice  of  A  rattier 
than  b. 

i‘o  help  us  distinguish  program  from  managerial  efficiency 
in  the  different  reference  sets  of  Dili's  we  shall  be  studying,  we  now 
-ntroduce  ctu  iolloving  extension  of  (2): 


,  ft  'i  a 

,a  h  u  y 

max  h  •  r-1  r  ro 
o - 

~  a  a  a 
h  v .  x  . 
r=l  i  10 


(6) 


subject  to 


a  a 

E,  u  y  .  , 

x  r  rJ  J  -  =1. 

—  m 

va  a  a 
.1.  v.  x. 
i=l  l  ij 


a 


a  a  .  „  ,  .  , 

u  ^  v  ^0 j  it  1 )  •  •  •  f  s  |  i~l )  •  i «  )  m  f 
r  i  a  a 


where  a=l,  2,  k, respectively,  indexes  the  sets  which  are  of  interest. 


Within  each  such  set  we  will,  of  course,  have  the  same  efficiency  measurement 
situation  as  before  —  viz.,  0  <_  h  a  j^l  with  hQa  =  1  if  and  only  if  the  DMU  being 
evaluated  relative  to  the  ath  set  of  DMUs  is  efficient .  Within  each  such  set  we 
shall  assume  that  we  are  securing  a  neasure  of  "managerial  efficiency."  Only 

when  allowance  has  been  made  for  the  presence  of  this  source  of  ineffi¬ 
ciency  will  we  be  in  a  position  to  a  ;sess  the  "program  efficiency"  that 
is  also  of  interest. 

In  order  to  illustrate  wha  is  involved  we  turn  to  data  from 
"Program  Follow-Through",  a  large -sc  de  experiment  in  U.  S.  public 
school  education  designed  to  help  di ^advantaged  children.  This  program 
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was  intended  to  further  the  objectives  of  a  pre-school  program  known  as  Project 
Head  Start,  by  carrying  on  specially  designed  programs  for  disadvantaged  chil¬ 
dren  from  kindergraten  through  grade  3.  This  was  done  at  the  70  selected 
"school  sites”  in  various  parts  of  tie  U.S.A.  which  are  listed  in  Table  A-l. 

(See  first  2  columns  of  Table  A-ll  fc t  the  numbering  that  will  hereafter  be  iden¬ 
tified  with  these  sites. 

An  assessment  of  Program  Follow- 7hro”gh  was  sought  in  what  was  intended  as 
a  designed  experiment  — ^  by  the  stud}  directors.  This  was  done  by  selecting  a 
matched  pair  of  Program  Follow-Through  and  Non-Follow-Through  sites  from  which 
observations  were  to  be  secured.  These  are  identified  as  PFT  and  NFT,  respec¬ 
tively,  in  columns  1  and  2  of  Table  A-l.  Although  we  need  to  observe  that  the 
experiment  was  not  carried  out  succef sf ully  in  all  respects,  it  nevertheless  pro¬ 
duced  a  wealth  of  data  from  which  we  effect  a  selection  as  follows. 

First,  we  restrict  ourselvjs  only  to  educational  aspects  of  the  program  wh i le 
ignoring  other  parts  of  its  activities  such  as  supplying  medical  and  dental  ser¬ 
vices,  nutritional  programs,  etc.  We  also  omit  any  explicit  consideration  of 
costs  on  the  supposition  that  a  cost/benefit  analysis  only  becomes  pertinent  if 
statistically  significant  effects  are  obtained  from  program  elements  such  as  we 
are  considering. 


—Or  at  least  as  close  to  such  an  experiment  as  one  is  likely  to  achieve  in  contem 
porary  U.S.  public  school  situations 


/O'  - 

Tu>  lc  A-l 


Site  L-’vel  Distribution  of  DFA  Study  Simple 


K^del  ar  J  Sit* 

Maa* 

naive  Education  Model 

t  rkeley.CA 
l  ffa *>,st 
0  lutK.rtv 
I  ••no ,CA 
L  j«non,NH 

S  it  lake.UT 
I*  coaa, WA 


Baltioofe.lO 

Lr  itvood|N'J 
Lla^ola.M 
Wichita  VS 

Bank  Street  Model 

K«v  York.SY 
Ft  lladel?  la, PA 
& • Jttlel >ro ,VT 
F.  II  River. KA 
V . 1 alngtcQ ,DC 


City  su* 


FFT  ' 

Student  Fop. 


Nn 

StuiUrat  Fop. 


17 

V'  v  York, MY 

ia 

62 

E  St.  Louts, IL 

19 

C  and  Faplds.MI 

20 

63 

F  eine.VI 

2i 

64 

I  lot.KI 

BA  H  del 

22 

M  *  York, ST 

2k 

6S 

1 1  lladelphla.PA 

74 

ft  rtagevillc.KO 

IS 

Kansas  City, MG 

26 

Loui  tvilif.n 

27 

Meridian, IL 

Cognitive  Curriculum  Kodel 

Vcv  Fork, KT 

CUcuo.ll 

Ok* loot*  Co. ,Fl 

Farent  Education  Model 


31 

FhlladelpMa.PA 

32 

66 

Jack  son vl  1 1  e , FL 

33 

67 

Ki  chnood , VA 

34 

68 

B.-uatoo.TX 

EDC  Model  . 

3S 

Pi»  lladel  phi  a.  FA 

36 

Facer aofl.KJ 

Self -Sponsored  Model 

37 

Detroit  ,Ml 

30 

69 

|rv  Yofk.KY 

39 

FMladelphla.FA 

40 

Frrrlaod.OR 

41 

Saa  DU|*,CA 

lUf  hod#  i 

42 

Krv  York, MY 

sent  H>d«l 

O  »  »  ten  cur  1» 

44  Idl.rt.CA 

Hom -School  F»fl»»r«M»  *"*' 1 

....  It  Uf|»  rtly  1* 

4)  krv  Tnrk.hT 

C.lff.rttl.  fror... 

«  " 

«»  trrrz-'*  »  »»«•*  ” 

:: _ ^r££c4 _ »  ” 

Tot* I  Stud'**  f»  r*  ^ 

•  fottlnd  fpyvl.t#  I  I  r  fl»u/r  tor 

**"  *  (Footnotes  eoiiliiineil  mi  next 


\ 

Medlua  City 

}  : 

C!  y 

t- : 

Hcdi'ia  City 

V 

KtdluB  City 

>£ 

Bural  Area 

\ 

Krdiu*  City 

t 

Hedlua  City 

Large  City 

i  l 

Soall  City 

1  C 

Medlua  City 

1C 

Ur|t  City 

1  E 

Urge  City 

1  E 

Large  Ctt> 

IE 

Sauill  City 

EE 

Medina  City 

S 

Mcdlua  City 

IK 

Urge  City 

SC 

Urge  City 

VC 

Medina  City 

VC 

hedluo  City 

MC 

Medina  City 

It 

Large  City 

re 

Ut|!  City 

VC 

Rural  Area 

VC 

l»t,»  City 

s 

Large  City 

VC 

Rural  Area 

Kt 

Ur|.  City 

tc 

Large  City 

s 

lull  City 

re 

Urg«  Clly 

s 

Large  City 

s 

large  City 

s 

l.tg.  CltJ 

IF. 

Larf.  City 

IE 

Mcdlua  City 

i : 

Large  Clly 

1  c 

Lire*  City 

%t 

Urge  City 

V 

Urge  City 

V 

Lst..  City 

sc 

Urge  Clly 

i : 

lai|«  city 

k 

Soall  City 

IE 

l,t|»  city 

l 

l«f,»  City 

l 

Saell  City 

» 

Rural  Area 

» 

Large  City 

31 

^4 

56 

21 

£ 

103 

- 

y* 

62 

27 

-■  £ 

77 

M 

A 
-  - 

ti 

- 

W 

ioa 

27 

<L 

47 

- 

'J 

61 

- 

90 

- 

6« 

32 

_ 

IS 

• 

4t 

' 

46 

13 

» 

111 

69 

93 

76 

112 

42 

‘ 

43 

20 

13 

86 

- 

43 

- 

71 

- 

•6 

M 

in 

26 

- 

98 

. 

74 

• 

27 

m 

42 

• 

26-  t- 


Table  A-  1 
(cont inuod) 


*NE  ■»  North  Eastern  United  States 
S  °  Southern  United  States 
NC  =  North  Central  United  States 
W  =  Western  United  States 


2 

Large  City 
Medium  City 
Small  City 
Rural  Area 


200,000  or  more 
50,000  to  199,999 
10,000  to  49,999 
Less  than  10,000 


3 

All  Data  Envelopment  Analysis  stud/  information  refers  to  the  Cohort  II-K 
student  population.  II-K  indicates  hat  this  group  of  students  began  their 
Program  Follow  Through  experience  in  kindergarten.  {This  was  also  the  only- 
one  of  three  Cohorts  which  had  completed  all  of  the  grades  from  kindergarten 
through  third  grade  at  the  tine  of  our  study  for  which  site  level  information 
was  available,)  However,  due  to  incomplete  statistics  along  some  DEA 
variable  dimensions,  some  of  the  Cohort  II-K  PFT  sites  were  not  included 
in  the  DEA  study.  Specifically,  Bank  Street  Model:  Rochester, NJ  site; 

EDC  Model:  Chicago ,1L  site;  and  SELL  Model:  St.  Maitin  Parish, LA  site 
were  excluded  from  the  DEA  study  student  population.  The  actual  Cohort  II-K 
PFT  population  was  3,3(7  of  which,  as  noted  above,  a  set  of  3,210  students 
were  used  in  the  DEA  study.  This  exclusion  of  sites  also  extended  to  the 
NFT  groups  which  were  similarly  reduced  to  1,202  students. 


4 

Two  se  ts  of  NFT  students  groups  were  created  in  the  original  Program 
Follow  Through  study.  One  group  was  a  local  student  set,  usually  in  the 
same  school  system  as  the  subject  PFT  site.  The  second  group,  and  the 
one  selected  for  the  DUA  study,  was  a  "best  matched"  group,  which  may  or 
may  not  nave  been  located  in  the  same  school  system  cr  even  the  same 
geographical  region.  The  NFT  group  which  most  nearly  matched  the  PFT  students 
of  a  given  site  along  a  number  of  demographic  and  initial  performance 
dimensions  was  considered  the  "best  match"  for  the  latter.  For  several 
PFT  sites  the  same  "best  matched"  NFT  group  was  used.  The  much  smaller 
NFT  student  population  total  of  1,202  as  compared  to  the  PFT  student 
total  of  3,210  resulted.  See  also  preceding  footnote. 
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*'e  shall  focus  on  only  one  of  several  cohorts  from  the  subjects 

handed,  in  the  study.  In  addition,  we  shall  utilize  only  the  terminal 

(grade  3)  results  for  this  cohort  to  avoid  the  additional  complications 

.■'ceded  zo  deal  with  dynamic  or  transient  behavior  en  route  to  this 

terminus,  from  a  set  of  11  output  measures  we  select  only  the  following 

3  as  -sufficiently  indicative  for  our-  purposes 

f  :  Tot;  i.  Reading  Score  as  measured  by  the  Metropolitan 
Achievement  Test. 

(7)  :  Total  Mathematics  Lcoia  as  measured  by  the  Metropolitan 

achievement  Test. 

:  Goo  )ersmith  self-esteem  Inventory,  intended  as  a  measure 
of  .elf-esteem. 

This  y 3  measure,  we  may  note,  is  directed  to  non-cognitive  growth  (or 
affective  behavior)  in  a  dimension  that  is  deemed  pertinent  to  the 
objectives  of  this  program.  Together  with  y^  and  ^2  'this  variable 
provides  a  good  indication  of  what  is  involved  in  assessing  such 
programs,  Note,  in  particular,  that  no  easily  available  scheme  for 
-..eighelng  the  relative  importance  of  these  outputs  is  at  hand.  Never¬ 
theless,  some  "overall"  measure  of  program  efficiency  is  wanted  in 
order  to  enable  us  to  evaluate  PPT  vi .  its  HPT  alternative  and  this  is 
to  be  echiived  from  data  such  as  are  exhibited  in  Tables  a-2  and  A-3. 
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Table  A-2 


Unadjusted  PFT  Output  Ohsi* rva t  ions 


Site  I  Total  Reading  Total  Hath  Total  Cooperooith 

Scores, PUS*  Scorefi.FHS*  Score*, PUS* 


1 

54,53 

58.98 

38.16 

2 

24.69 

33.89 

26.02 

3 

36.41 

40.62 

28.51 

4 

14.94 

17  58 

16.19 

5 

7.81 

6.94 

5.37 

6 

12.59 

16.85 

12.84 

7 

17.06 

16.99 

17.82 

8 

20.29 

30.64 

33.16 

9 

26.13 

29.80 

26.29 

10 

46.42 

51.59 

35.20 

n 

39.80 

37.73 

30.29 

12 

37.84 

47. 8S 

25.35 

u 

26.48 

31.36 

26.54 

u 

10.31 

10.86 

7.47 

IS 

14.39 

18.30 

14.33 

16 

32.94 

36.03 

38.19 

17 

17.25 

20.80 

12.07 

18 

27.55 

38.19 

20.44 

19 

41.12 

43.80 

36.54 

20 

29.43 

42.63 

23.34 

21 

37.46 

51.02 

27.44 

22 

19.40 

25.18 

16.52 

23 

39.88 

47.72 

38.97 

24 

25.72 

30.81 

16.54 

25 

24.88 

25.27 

22.43 

26 

31.62 

40.78 

31.16 

27 

31.31 

38.32 

25.03 

28 

21.00 

21.30 

18.30 

29 

6.51 

7.02 

6.16 

30 

11.64 

15.26 

15.68 

31 

12.58 

15.90 

14.42 

32 

4.59 

6.16 

4.99 

33 

43.76 

46.64 

39.10 

34 

32.38 

38.55 

31.05 

35 

34.64 

45.46 

39.22 

36 

11.52 

15.14 

13.91 

37 

15.96 

19.21 

15.30 

38 

9.91 

12.30 

7.22 

39 

30.44 

33.53 

29.80 

40 

22.63 

25.24 

17.15 

41 

24.41 

27.16 

25.30 

42 

23.11 

22.67 

17.56 

43 

21.82 

31.45 

27.54 

44 

63.92 

79.67  • 

63.11 

45 

9.47 

11.92 

8.85 

46 

33.94 

39.18 

34.61 

47 

29.42 

35.10 

28.42 

48 

7.70 

11.02 

9.02 

49 

12.17 

16.03 

15.82 

•  fits  •  Per  Hundred  Students 
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able  A-3 


Unadjusted  NFT  Output  Observations 


Sice  t 

Total  Reading 

Sc )tes>?HS* 

Tot  1  Hath 

See  ea.PHS* 

Total  Coope 
Scores ,PHS* 

Yi 

T2 

T3 

30 

.9.07 

4?.  71 

27.67 

51 

9.96 

1<  .34 

9.33 

52 

45.37 

51.38 

31.61 

53 

18.23 

22.05 

17.56 

54 

59.63 

64.41 

35.89 

55 

24.20 

28.21 

18.74 

56 

13.53 

17.09 

15.61 

57 

28.39 

27.65 

20. 79 

58 

21.67 

26.22 

13.66 

59 

•  120.17 

144.67 

88.59 

60 

15.15 

18.04 

13.58 

61 

6.92 

7.10 

6.35 

62 

9.35 

9.85 

7.70 

63 

13.03 

13.40 

10.29 

64 

18.63 

24.48 

23.13 

65 

12.28 

13.01 

9.89 

66 

16.81 

19.72 

18.70 

67 

26.36 

28.22 

24.46 

68 

22.85 

26,21 

28.14 

69 

8.17 

8.70 

5.12 

70 

13.69 

14.19 

12.99 

•ms  * 

Per  hundred  Students 

•*>%,  >««» 


T 
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Given  these  output  variables  ve  next  turn  to  a  choice  of  the  inputs  to  be  used 
in  our  study.  From  data  on  the  25  input  variables  utilized  in  the  Follow-Through 
study  we  have  selected  the  following  5  for  purposes  of  our  illustration: 


X1  :  education  level  of  nt'ther  as  measured  in  terms  jf  per¬ 
centage  of  high  seho >1  graduates  among  fern;  le  parents. 

:  nighest  occupation  o.  a  family  member  according  to  a 
pre-arranged  rating  ;.cale. 

x3  s  Parental  visit  index  representing  the  number  of  vi3its 
to  the  school  site  o  .•  with  Pol  low- Through  personnel. 

:  Parent  counseling  index  calculated  from  data  on  time 

spent  with  child  on  :chooi  related  topics  such  :  s  reading 
together,  etc. 

s  Number  of  teachers  a-  a  given  yi'l’  or  HPT  site. 


The  data  for  these  variables  from  both  PFT  and  NFT  sites,  respectively,  are  ar 
ranged  in  Tables  A-4  and  A-5. 


UilS  TJJSX  IS  5TS-T 
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Table  A-i; 


Unadjusted  PFT  In  out  Observations 


lit*  t 

Education  Le  \ e  1 

Occupat ion 

Parental  7isit 

Counseling 

of  Mother, PHi* 

Index, PHS* 

Index, PUS* 

Index, PUS 

V 

*2 

X3 

X4 

1 

36. 13 

16.24 

48.21 

49.69 

2 

29.26 

10.24 

41.96 

40.65 

3 

.3.12 

11.31 

38.19 

35.03 

4 

!4.96 

6.14 

24.81 

25.15 

5 

11.62 

2.21 

6.85 

6.37 

6 

1.88 

4.97 

18.73 

18.04 

7 

<2.64 

6.88 

28.10 

25.45 

8 

10.79 

12.97 

54.85 

52.07 

9 

34.40 

11.04 

38.16 

42.40 

10  . 

61.74 

14.50 

49.09 

42.92 

11 

52.92 

11.67 

39.48 

39.64 

12 

36.00 

10.15 

37.80 

39.52 

U 

39.20 

10.80 

41.04 

41.12 

14 

14.6 

2.88 

9.64 

11.14 

15 

4.29 

5.42 

21.45 

17.27 

16 

47.25 

14.17 

56.46 

55.26 

17. 

7.2.63 

4.43 

15.40 

15.00 

18 

78.00 

7.61 

28.73 

27.04 

19 

:  3.56 

13.70 

53.04 

49.85 

20 

:  5.42 

9.05 

29.69 

31.74 

21 

31.57 

10.08 

39.34 

40.57 

22 

16.34 

5.84 

20.89 

22.10 

23 

44.28 

14.14 

56.70 

52.27 

24 

39.74 

6.43 

24.20 

25.66 

25 

24.40 

8.05 

33.42 

31.29 

26 

41.40 

11.70 

44.01 

46.35 

27 

27.20 

9.38 

37.80 

31.55 

28 

23.92 

7.12 

25.58 

29.01 

29 

10.62 

2.55 

10.10 

9.09 

30 

12.48 

6.14 

23.13 

22.46 

31 

19.32 

5.89 

24.01 

24.74 

32 

6.30 

1.93 

7.11 

7.68 

33 

46.62 

14.65 

65.71 

57.49 

34 

38  95 

12.82 

47.02 

48.92 

35 

oj  •  50 

15.56 

53.98 

50.29 

36 

31.08 

6.26 

22.18 

21.96 

37 

19.35 

6.68 

22.61 

23.31 

38 

11.20 

3.08 

9.90 

10.06 

39 

34.40 

11.61 

41.79 

41.79 

,0 

35.55 

6.48 

21.69 

21,69 

41 

30.53 

9.30 

35.50 

35.14 

.2 

25.44 

7.10 

26.81 

26.23 

,3 

76.66 

11.43 

41.36 

44.63 

.4 

39.79 

22.49 

84.77 

76.12 

45 

8.32 

3.64 

12.92 

13.13 

46 

59.73 

13.52 

48.80 

49.69 

47 

39.22 

10.06 

37.00 

38.33 

48 

3.28 

3.18 

13.12 

12.71 

49 

7.14 

3.29 

23.10 

19.06 

Number  >f 
Teachr i  j 


9 

5 

9 

7 

4 

4 

7 

8 
8 
9 

5 
S 
7 

3 

5 
9 
2 
9 
7 

4 

6 
4 

11 


11 

2 

13 


•rilS  *  Per  Hundred  Student* 


l\i  b  1  e  A-b! 


llnatl juste .J  NFT  Input  OIjsc rv.i tlotifi 


Site  # 

Education  Level 
of  Mother, PUS* 

Occupation 

Inde*tPUS* 

Parental  Visit 

Index ,  PHS* 

Counseling 
Index, PHS* 

Nusjber 

Teachei 

X1 

*2 

X3 

X4 

X5 

SO 

68.16 

12.28 

33.58 

34.64 

15 

51 

11.88 

3.59 

13.41 

13.82 

e 

52 

55.30 

11.53 

36.73 

35.78 

t 

53 

16.20 

7.02 

26.94 

26.30 

9 

54 

82.45 

15.52 

45.00 

44.23 

13 

55 

15.81 

6.93 

23.91 

23.61 

7 

56 

4.65 

5.50 

20.91 

23.39 

5 

57 

41.25 

8.41 

26.23 

25.24 

10 

58 

10.44 

5.22 

17.10 

18.93 

3 

59 

139.65 

35.03 

119.56 

130.83 

22 

60 

16.28 

4. PI 

18.20 

18.98 

5 

61 

12.06 

2.59 

8.74 

8.17 

5 

62 

4.20 

2.64 

9.89 

11.25 

2 

63 

19.44 

3.83 

12.87 

13.23 

5 

64 

28.38 

8.91 

30.95 

33.33 

8 

65 

13.50 

3.61 

15.60 

12.39 

4 

66 

23.32 

7.10 

24.96 

28.56 

22 

67 

27.60 

9.38 

32.29 

34.01 

20 

68 

11. 70 

10.53 

37.67 

43.60 

8 

69 

4.68 

1.85 

6.22 

5.46 

5 

70 

10.44 

4.82 

17.13 

18.21 

9 

•PUS  m  Per  Hundred  Students 
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Given  che  preceding  data  we  cry  next  to  d isentangle  any  managerial  Inef¬ 
ficiencies  chat  might  otherwise  appear  as  program  inefficiencies.  This  is  done  by 
assigning  a  -  1  to  PFT  and  a=  2  to  NFT  n  (6)  and  separately  calculating  the  re- 

*  X 

.pective  h  values  that  are  exhibited  in  Table  A-6.  In  each  case  the  efficient 
r  o 

nembers  of  the  set  generate  an  "envelope,"  or  efficiency  frontier,  which  we  shall 
refer  to  as  a- envelopes .  Thus,  e.g.,  a  =  1  refers  to  the  envelope  for  PFT  and 
i-2  refers  to  the  envelope  for  NFT  vfiich  we  shall  also  label  as  the  "1-envelope" 
ind  "2-envelope,"  respectively. 

■  )ne  question,  we  may  now  ask  is  whether  the  managerial  effi¬ 
ciencies  are  the  same  for  both  seta  of  observations  or  whether  the  results 
of  a  frT-iN.'T  comparison  might  be  clouded  by  difference:  in  this  very 
important  dimension.  Ideally,  this  should  have  been  allowed  for  in  the 
statistical  design  but,  of  course,  it  would  be  difficult  to  obtain  the 
needed  d_ta  on  efficiencies  in  advance.  However,  we  can  at  least  make 
allowances  and/or  effect  adjustments  for  this  along  lines  like  the 
following. 

For  instance,  we  might  calculate  the  proportion  of  managers  who 
*« 

achieve  values  of  hQ  =1  under  at  =1  and  cc  =  2,  respectively.  This  gives 


(9) 


P(ct  =1)  =  me(1)  =  17  =  Sl_ 
m  1)  ^9  1UT 

*1 


where  me(l;  is  the  number  of  PFT  flt-IU  s  with  hQ  values  of  1.0  and  m(l) 
is  the  number  of  DMU's  in  PFT.  Simi.  arly  for  NFT  we  obtain 


(10) 


P(c<  =2)  =  =  _ 8_  =  _ 56 

n  (2)  21  11+7 


where  m(a)  refers  to  the  total  number  of  EMU' s  in  IJFT  and  “e(2)  refe^ : 
to  the  number  on  the  efficiency  frontier. 
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Tabic  A-6 


PIT  and  NFT  Program  Specific  o-Envelope  Efficiency  Values 


PFT 
Site  # 


*1 

h0 

Efficiency  Value 


*2 

NFT  h0 

Site  §  Efficiency 


1* 

1.00 

2 

0.90 

3 

0.98 

4 

0.90 

5* 

1.00 

6 

0.90 

7 

■  0.89 

8 

0.91 

9 

0.87 

10* 

1.00 

11 

0.98 

12 

0.97 

13 

0.86 

14 

0.98 

15* 

1.00 

16 

0.95 

17* 

1.00 

18* 

1.00 

19 

0.95 

20* 

1.00 

21* 

1.00 

22* 

1.00 

23 

0.96 

24* 

1.00 

25 

0.97 

26 

0.93 

27* 

1.00 

28 

0.94 

29 

0.84 

30 

0.90 

31 

0.83 

32 

0.90 

33 

0.94 

34 

0.85 

35* 

1.00 

36 

o.'eo 

37 

0.94 

38 

0.94 

39 

0.91 

40* 

1.00 

41 

0.94 

42 

0.94 

43 

0.87 

44* 

1.00 

45 

0.89 

46 

0.90 

47* 

1.00 

48* 

1.00 

49* 

1.00 

50 

0.95 

51 

0.92 

52* 

1.00 

53 

0.87 

54* 

1.00 

55* 

1.00 

56* 

1.00 

57 

0.92 

58* 

1.00 

59 

0.92 

60 

0.98 

61 

0.88 

62* 

1.00 

63 

0.96 

64 

0.91 

65 

0.97 

66 

0.92 

68* 

1.00 

69* 

1.00 

70 

0.94 

*  Denotes  a  site  with  an  efficiency  v  iluo  of  "1* 


Value 
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Proceeding  row  in  a  somewh  it  informal  manner,  we  can  see  tn_i 
che  managers  in  the  two  programs,  PF  1  and  NF*T,  have  ^bout  an  equal  like¬ 
lihood  of  being  on  their  respective  jrogram  referenced  efficiency  frontiers. 

The  differences  betw  en  p(o  =1}  and  ?(ot  =2)  are  not  statistically  signifi¬ 
cant  co  ti  at  an  impl  .cation  of  the  a  love  results  is  that  both  PFT  end  r,.  i1 
site  man.,;er^  or  "decision-makers*  a  -e  drawn  from  the  same  managerial 
efficiency  pool  or  population.  -  a  .ively  speaking  there  appear  to  be 
just  as  cu-ny  managers  in  PFT  as  in  N'T  who,  within  the  3 imits  of  their  pro¬ 
gram  constraints,  operate  their  sites  efficiently. 

The  above  probability  measiires  provide  insight  only  into  the  relative 

location  of  DMJ's  in  PFT  and  NFT  with  respect  to  their  boundaries. 

Questions  involving  differences  between  the  distribution  of  efficiency 

values  in  the  two  groups  ire  not  addressed  by  the  above  calculation.  Thus 

to  check  md  perhaps  extend  our  understanding  of  these  results,  while  still 

staying  with  relatively  simple  measures,  we  consider  e  comparison  between 

the  mean  efficiency  value  differences  in  the  two  sets,  again  using  the 

information  contained  in  Table  2,  we  compute  a  PFf  mean  efficiency  as 
-*1  „  -*2 

hQ  =  O.S»l+6  and  an  NFT  mean  efficiency  value  as  hQ  =  0.958.  Thus,  NFT 

has  a  slightly  higher  mean  efficien  ;y.  but  again  the  difference  is  not 
statistically  significant. 

having  arrived  at  these  re  ;ults  we  are  at  the  point  of  choosing 
whether  to  elect  a  "pure  prediction"  or  a  "control  prediction"  course 
along  the  lines  that  we  indicated  wh  n  discussing  Figure  2.  Proceeding 
long  the  latter  lines  we  might  arri/e  at  a  situation  such  as  the  one  shown 
in  Figure  3.  here  the  envelopes  fo  ’  PFT  and  NFT  cross  over  from  one  part 
of  the  diagram  to  another  which  mean  j  that 

PFT  is  more  efficient  in  the  region  to  the  right  and  NFT  is  more 
efficient  in  the  left-hand  pert  ion  if  the  diagram. 
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Analyses  could  be  centered  on  tach  of  the  latter  segments,  of  course,  but 
something  more  might  he  wanted.  We  night,  in  particular,  want  an  overall  rssess- 


ment  to  enable  us  to  choose  between  the  two  programs  and  for  this  we  might  proceed 


as  follows.  First,  we  might  re  place  the  or iginally  observed  output  values  y  and 


2  12 

y  .  with  new  values  v  .  and  y  .  which  result  when  the  orieinallv  observed  values 
rj  '  rj  ri 


are  adiusted  in  a  manner  that  brine-  them  onto  their  respective  envelopes  .^Similarly 


1  2 

we  can  replace  the  original  and  input  '--alues  in  Tables  A-4  and  A-5  by  new 

-  1  2 

values  x..  and  x..  that  are  derived  from  their  efficiency  frontiers.  Thenwecan 
ij  1J 


derive  t  new  envelope  which  we  shall  refer  to  as  the  ''inter-envelope,"  via 


£  u  y 

r  —  1  r- 


max.  h  = 
o 


U  J 

r=l  r  ro 


.X,  v.  x 
1=1  1  10 


subject  to 


au 


1  > 


I.  u  y  . 
r=l  r  n 


j  =  l. 


,  m 


. I .  v.  x. . 
i=l  l  ij 


and 


1  > 


,  U  v  . 

r=l  r _ rj 


j  =  l. 


—  ra 


i£i  vi  xi 


ow  beior*:3^  .11  varxa.  le  v-  .uea  coni  ir.  nod  to  b~  pot  iU’A  . 


—  The  manner  of  doing  this  is  given  n  p]  and  [i'3]. 
-^Supra  ( 2 ) f f . 


r  L 


5 


X 


id; 

0  *  k>FT  Observations 

X  *  NFT  Observations 

,  a  Envelope  for  PET 
=>  <»  Envelope  for  NET 
....  =,  Inter-envelope 
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Application  of  (11)  produces  the  results  presented  in  Table  A- 7,  as  well  as  the 

remaining  results  needed  for  the  expl ication  of  Figure  3.  Note  that  we  are  nov  assum- 
int  that  ail  managers  are  operat ing  el f iciently,  by  which  we  mean  that  they  will  al¬ 
ways  move  to  the  boundaries  indicated  by  their  respective  envelopes.  We  similarly  as¬ 
sume  that  they  will  move  to  any  new  boundaries  that  may  become  available  when  their  pre¬ 
sent  confinement  to  either  PFT  or  NF1  boundaries  only  is  relaxed,  and  this  is  what 
the  inter-envelope  comparison  s  inte  nded  to  p:o\,  ide.—^  Notice,  for  example  that 

the  inter-envelope  is  always  as  efficient  as  either  the  a  =  1  or  the  a  =2  envelope, 
and,  in  fact,  it  coincides  with  them  i  hen  one  or  the  other  is  more  efficient. 

All  that  remains  now  in  effecting  our  evaluation  is  to  select  a  suitable 

*  * 

statistic  that  will  reflect  how  the  h^  values  are  distributed  for  a  = 1  and  i  =  2 

relative  to  their  potential  values  when  the  present  boundaries  are  removed.  This 

2/ 

is  provided  by  the  Kullback-Leibler  statistic—  for  which  we  get 
I  (cC»l)  -  2.1*0226  x  10~k 

and 

I  (oc  =2)  =  0.03651*  x  10~U. 

xhic  means  that  the  distribut  ion  for  the  adjusted  PFf  observations,  taken  all  together.are 
more  distant  from  the  inter-envelope  than  is  the  case  for  the  distribution  of  the 

3/ 

Icn  observations.-  In  othex  words  PIT  Fails  the  overall  efficiency  tost  at 
1  st  for  the  cohort  und.  the  variables  that  we  nave  ex  mined. 

-^More  precisely,  we  are  assuming  that  these  managers  can  be  brought  onto  the  boun¬ 
daries  —  while,  of  course,  we  are  also  providing  estimates  of  the  losses  (or  wastes) 
that  will  be  experienced  when  this  is  not  done.  This  is  to  say  that  our  adjustments 
are  enforcing  these  assumptions  for  ail  DMU's  although,  of  course,  we  need  not  always 
proceed  in  this  manner.  We  may  in  fact  restrict  our  adjustments  only  to  subsets  cf  the 
DMU's  and/or  make  only  partial  adjustment  of  designated  inputs  or  outputs  when  desired. 

-Uee  [14]. 

3/ir 

—  On  the  Statistical  Distribution  of  DMU  Efficiency  Measures"  b'  A.  Charnes 
W.  W.  Cooper  and  E.  Rhodes  in  [8]  provides  a  canonical  d jvelopment  which  is  il¬ 
lustrated  by  a  direct  PFT/NFT  comparison.  This  comparison,  which  does  not  util¬ 
ize  the  inter-envelope,  yields  the  sar e  conclusion  —  viz.,  the  data  and  criteria 
used  in  this  paper  do  not  show  PFT  to  be  more  efficient  than  NFT  in  any  statisti¬ 
cally  significant  sense. 
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Table  a-7 


Inter-Envelope 

Efficiency  Values 

PFT 

* 

* 

ho 

NFT 

h0 

Site  # 

Efficiency  Value 

Site  # 

Efficiency 

1 

0.92 

50* 

1.00 

2* 

1.00 

51* 

1.00 

3 

0.94 

52* 

1.00 

4* 

1.00 

53* 

1.00 

5 

0.93 

54* 

1.00 

6* 

1.00 

55 

0.99 

7 

0.99 

56* 

1.00 

8* 

1.00 

57* 

1.00 

9 

0.98 

58* 

1.00 

10 

0.92 

59* 

1.00 

11* 

1.00 

60 

1.00 

12* 

1.00 

61* 

1.00 

13 

0.99 

62* 

1.00 

14 

0.95 

63* 

1.00 

15* 

1.00 

64* 

1.00 

16* 

1.00 

65* 

1.00 

17* 

1.00 

66* 

1.00 

18* 

1.00 

67* 

1.00 

19 

0.99 

68 

0.99 

20* 

1.00 

C9* 

1.00 

21* 

1.00 

70* 

1.00 

22* 

1.00 

23 

0.99 

24* 

1.00 

25* 

1.00 

26 

0.99 

27* 

1.00 

28* 

1.00 

29 

0.99 

30* 

1.00 

31 

0.99 

32* 

1.00 

33 

0.99 

34 

0.98 

30* 

1.00 

36* 

1.00 

37 

0.94 

38 

0.99 

39* 

1.00 

40 

0.95 

41 

0.99 

42* 

1.00 

43 

0.99 

44* 

1.00 

45 

0.99 

46* 

1.00 

47* 

1.00 

48* 

1.00 

49* 

1.00 

Value 


•Denotes  a  cite  with  .in  efficiency  value  of  "1 
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Of  course, this  need  not  end  th<  matter  since,  as  we  have  already  noted,  a 
facet-by  facet  comparison^  might  yie  d  situations  in  which  PFT  is  superior  to 

1The  basic  sets  assoicated  with  the  simplex  (and  dual  method)  calculations  as 
described  in  [  4  ]  and  [5]  will  make  it  easy  to  distinguish  the  various  facets. 

NFT.  In  addition,  the  further  possit ilities  associated  with  the  broken 
line  segment  shown  for  the  inter-env£ lope  in  Figure  4  might  also  be  noted.  It 
will  be  observed  that  this  segment  if  not  coincident  with  either  of  the  original 
i  envelopes.  The  reason  is  that  thi:  segment  contains  "mixture"  possibilities 
that  are  not  apparent  from  either  em elope  alone.  That  is,  it  represents  situa¬ 
tions  in  which  the  PFT  and  NFT  programs  might  be  combined  to  yield  still  further 
possibilities  for  improvement.  In  other  words,  our  analysis  has  not  only  yielded 
the  wanted  program  comparisons  but  it  has  also  yielded  a  fLexible  way  of  effecting 
more  detailed  (facet  by  facet)  comparisons  as  well  as  entirely  new  program  possi¬ 
bilities  in  terms  of  one  analytical  approach. 

5.  COICLUSION 

Appropriate  conclusions  will  be  drawn  from  the  discussion  of  this  paper  at  the 
Hawaii  TIMS  meetings  and  entered  in  a  revised  version  of  this  paper.  The  latter 
will  be  sent  to  interested  persons  in  response  to  requests  sent  to  either  author. 

In  the  meantime  we  also  herewith  supply  an  appendix  which  provides  a  characteriza¬ 
tion  which  is  canonical  for  the  kinds  of  statistical  distribution  involved  along 
with  an  illustration  of  their  use  when  direct  PFT  vs  NFT  comparisons  instead  of 


the  comparisons  effected  through  the  inter-envelope  of  Figure  3  are  wanted. 


ADDEND!  !M 


We  might  better  point  the  concluding  section  of  this  paper  into  possible 
new  directions  as  follows-  After  the  above  plenary  session  paper  was  presented 
at  the  TIMS  Meetings  in  Honolulu,  Hawaii,  Alan  Hoffman^  of  IBM  observed  to  us 


See  [20]  for  comments  on  Dr.  Hoffman's  contributions  to  the  efficiency  studies 
of  M.  J.  Farrell  at  an  earlier  date. 


that  the  r  onceots  in  our  paper  might  be  used  to  study  comparative  social-economic 
systems,  n  which  event  one  might  find  that  the  Russian  (socialist)  system  was 
better  but  failed  to  show  this  because  it  was  more  poorly  managed  than  the  U.S. 
(capitalist)  system.  Our  reply  was  hat  if  such  a  study  were  conducted,  its  au¬ 
thors  had  better  be  aware  of  the  difference  between  control  predictions  and  pure 
predictions  made  elsewhere  in  this  paper. 

An  opportunity  to  explore  those  ideas  further  was  provided  in  a  post-meet¬ 
ings  Radical  Workshop  on  Organization  Design  organized  by  Arie  Lewin  and  Kenneth 
MacKenzie  on  the  island  of  Maui.  One  whole  morning  being  set  aside  at  this  Work¬ 
shop  for  the  organization  design  implications  of  our  efficiency  measures  suggested 
a  more  modest  route  to  us  than  the  undertaking  mentioned  by  Dr.  Hoffman.  In  parti¬ 
cular,  we  were  aware  that  a  great  deal  of  the  small  group  work  undertaken  by  Dr.  Mac - 
2/ 

Kenzie—  in  the  field  of  organization  design  has  often  been  concerned  with  multipl 

2 

See,  e.g.,  K.  D.  MacKenzie,  A  Theory  of  Group  Structure  —  Vol.  I,  Basic  Theory 
and  Vol.  II,  Empirical  Tests  (New  York,  Gordon  &  Breach,  1976). 
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outputs.  To  be  sure,  such  analyses  are  generally  confined  to  only  a  very  few 
output  measures  -  such  as  (1)  time  to  complete  a  task  with  different  kinds  of 
organization  arrangements  and  (2)  degree  of  satisfaction  with  the  work  by  persons 
in  various  parts  of  each  type  or  organiz. ■  t  ion  --  and  almost  never  involve  more  than  a 
few  input  measures  such  as  time  and  numb  ;r  of  persons.  On  the  other  hand,  these 
data  do  provide  a  start  and,  of  courts,  our  efficiency  measure  should  provide 
a  basis  for  extension  into  more  complex  input  and  output  considerations  in 
the  future. 

A  systematic  basis  for  further  progress  in  this  work  may  thus  be  pro¬ 
vided  that  could  then  be  extended  to  the  study  of  efficiency-effectiveness  in¬ 
teractions  —  and  perhaps  other  new  dimensions  for  research  might  thereby  also 
be  opened.  The  point  on  which  we  can  now  conclude  is  that  program  vs.  manag¬ 
erial  efficiency  or  even  organization  vs.  managerial  efficiency  under  different 
programs  are  not  the  only  possibilit  _es.  What  is  required  for  the  immediate  fu¬ 
ture  is  some  meaningfully  measurable  way  of  identifying  different  sources  of 
efficiency,  and  laboratory  experimentation  along  the  dimensions  of  small  group/or¬ 
ganization  design  research  certainly  offers  an  inviting  prospect  for  such  possi¬ 


bilities. 
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Introduction 


The  discussion  in  the  text  was  directed  to  distinctions  be¬ 
tween  program  eff  iciencies  and  managerial  ef  f  ie  ienc.  i es  and  ways  in  which  the  two 
might  be  separated  for  purposes  of  program  evaluation  and  control.  These 
topics  aie  sufficiently  novel  and  complex  so  that  it  seemed  best  to  focus 
on  tnem  without  diverting  attention  to  still  further  possibilities. 

One  such  additional  alternative  that  needs  to  be  considered  in¬ 
volves  the  case  in  which  adjustments  to  achieve  improved  efficiency  in 
management  practices  cannot  (or  will  not)  be  made.  Then  the  evaluation 
must  be  made  along  the  lines  that  we  have  characterized  as  the  "pure  predic¬ 
tion"  situation  (in  which  it  is  assuned  that  management  practices  vill  con¬ 
tinue  with  the  same  mix  of  efficien< y  levels  as  before). 

Undoubtedly  still  other  alternatives  might  need  to  be  considered. 
Witness,  for  instance,  the  situation  for  energy  studies  where,  as  described 
in  the  text,  only  adjustments  for  ei ergy  consumption  and  related  uses 
might  be  of  interest  on  the  assumption  that  other  parts  of  the  relevant 
DMU  operations  will  subsequently  be  adjusted  to  these  more  efficient 
methods  of  energy  utilization.  In  iny  case  the  unusual  character  of  the 
statistical  distributions  being  considered  makes  it  desirable  to  establish 
canonical  references.  If,  for  example,  we  wish  to  employ  the  Kullback- 
Leibler  (or  "minimum  discrimination  information")  measures  of  discrepancy, 
as  in  the  text,  we  must  compare  on  the  basis  of  distributions  that  are 
mutually  absolutely  continuous.1  A  simple  way  to  affect  this  is  to  estimate 
for  each  example  the  parameters  of  a  "canonical  distribution"  from  a  class 
which  has  this  mutual  property. 


The  remainder  of  this  appendix  is  therefore  pointed  toward  the 
development  of  such  a  canonical  class.  We  do  this  in  a  way  that  relies  on 
fundamental  properties  of  the  gamma  distribution  since  the  latter  appears 
appropriate  to  the  non-negative  measurement  scales  of  the  inputs  and  outpits. 


Background 


The  diagrams  shown  in  the  Figures  at  the  end  of  this  appendix, 
which  are  drawn  from  the  data  of  Table  A-6  in  the  text,  will  serve  as  back¬ 
ground  for  the  developments  we  shall  now  introduce.  We  ma  /  think  of  these 
as  figures  derived  in  the  following  manner. 

Each  program  sample  of  DMU's  determines  a  vector  of  sample  effi¬ 
cient  ;s  ,  one  for  each  DMU.  These  efficiency  ratings  are  ibtained  from  a 
ratio  of  linear  functionals  which  is  optimized  over  a  poly  ledral  constraint 
set  that  is  the  same  for  every  DMU. 

To  make  the  sense  of  this  more  concrete  we  write  our  sample  data 
for  a  particular  program  in  the  form 

(X1,  Y1),  ...,  (Xj,  Yj),  ....  (X°,  Y°) 


where  each  X,Y  pair  corresponds  to  a  vector  of  input  and  output  observations 

for  one  of  j=l,  n  DMU's  in  that  program.  The  efficiency  for  the  jth 

DMU  is  then  obtained  via 

*  uTYj 

h,  =  max  —  t 

J  v  XJ 


T  1  T  1 

subject  to  u  Y  -  v  X  <  0 


uTYj  -  vTXj  <  0 


Twn  Tvn 
u  Y  -  v  X  <0 

T  T 

with  u  ,  v  >Q  wherein  the  T  superscripts  indicate 


the  t  anspose  of  the 


corresponding  column  vector. 
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i 


Now  let  s  be  the  slack  vector  with  non-negative  component  Sj>  . . . , 

Sj ,  s^,  which  can  be  associated  with  the  above  inequalities  to  bring 

them  into  equation  form.  Then  we  can  observe  (i)  that  there  are  a  finite 

k  lc  k 

number  of  ext  erne  points  (u  ,  v  ,  s  )--  k=l ,  k  --which  span  this  con- 

k  k  k 

straint  set  and  (ii)  the  optimal  h  (X,Y)  is  attained  for  some  (u  ,  v  ,  s  ) 
We  therefore  have  the  following  thee rems  at  our  disposal: 


Theorem:  h^  (X,Y)  =  1  if  s^  =  1  for  some  extreme  point 

,  k  k  k. 

(u  ,  v  ,  s  ). 


Theorem:  In  every  sample  h.  (X  Y)  =  1  is  attained  for  at 

least  one  j . 


Thus  the  population  density  function  for  efficiency,  arising  from  the  popula¬ 
tion  from  which  the  program  sample  of  n  DMU’s  and  the  sample  (XJ  ,  YJ )  for 
each  j  is  drawn  ,  must  have  a  part  at  efficiency  =1.  We  shall  refer  to 
this  as  the  "atomic  part"  and  assunu  in  the  developmei  t  which  follows  that 
the  efficiency  density  function  is  a  mixture  of  a  cont.inuous  density  and 
the  atomic  density  at  t=l.  See  Figures  at  the  end  of  this  appendix. 


Density  Characterizations  and  Developments 


We  proceed  now  to  choose  a  class  of  continuous  density  functions 
from  the  following  considerations:  First,  the  vectors  X^  ,  are  assumed 
to  be  vectors  of  sample  averages,  one  average  for  each  input  and  output  of 
each  DMU.  These  averages,  a  priori,  are  non-negative.  If  we  assume  that 

each  average  is  from  a  gamma  distribution  of  the  form  g^  (t)  =  \P  tp  1  e  ^ 

k  k  k  k 

then  so  is  each  u  y  .  and  each  v.x. .  where  these  u  and  v.  are  components  of 
r7rj  l  ij  r  i 

k  k 

the  positive  vectors  u  and  v  .  Assuming  further  that  these  averages  are 


statistically  independent  of  one  another1  and  that  they  have  the  same  ultim¬ 


ate  "decay"  rate  then  a  numerator  N  (say  1.  u  y  . )  and  its  denominator  D 

k  r 

(say  2  v  p  )  specifying  an  efficiency  rating  are  each  independently  gamma 

r  1  1 J 


distributed  with  parameters,  (p,A)  and  (q,A),  respectively. 
Now2  the  density  function  for  N/D  is 


h( 


„  4 


r(P)  r (q)  tp  *(i+t)  p  q 


t  >  o 


,  elsewheri 


where 


f(s)  =  f  ts  *e  tdt  , 


.  f(s)  =  (s-l)I  for  s  integer 


R  R 

But,  further,  our  constraints  require  that  N/D  =  2  u  y  ./I  v.x. .  <  1.  Thus, 

1  rrj  .in-  ’ 

r  J  i  J 


the  density  function  h(t)  for  N/D  conditional  on  N/D  <  1  is 


1 


h(t) 


h(t)//h(s)ds,  0  < t  <  1 

o 


0 


elsewhere 


We  note  that  the  mode  of  the  numerator  is  at  x^  =  (p-l)/\,  and 
that  of  the  denominator  is  at  x^  =  (q-l)/\,  so  that  the  mode  of  h(x)  — 
and  also  that  of  h(x) —  is  at 

x  =  (p-l)/(q+l), 

i.e.,  for  large  p  and  q  is  approxima  ely  at  the  ratio  of  the  modes.  From 
the  empirical  fact  that  most  operations  are  better  than  say  75%  or  80% 
efficient,  we  can  expect  from  the  fu  ictional  form  for  h(x)  that  (i)  the 
values  of  p  and  q  will  be  quite  large,  (ii)  that,  p-1  <  q+1 ,  and  (iii)  that 
p  and  q  are  about  the  same  magnitude. 
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Because  of  these  large  powers,  P  and  q,  it  is  not  convenient  to 
carry  out  integrations  to  determine  h(x) .  Thus  to  estimate  p  and  q  from 
sample  data,  we  have  fit  the  sample  relative  frequency  function  h(x)  (non- 
atomrc  part!  as  follows: 

We  take  the  sample  mode  t<  be  x,  Substituting  p-1  =  x(q+l),  we 

have 

h(x)  =  x(q+1)5(ltx)-(^l)(1+i)  ,  0  <  x  '  1  . 

1  2 

Then,  considering  two  values  x  ,  x  ,  we  get 


Therefore , 


hex1)  =  h(xb  =  ii+A-wni*) 

h(x2)  h(x2)  \x2J  V*2) 


=  (q+1) 


(1+x)  In 


In  our  case,  since  (a)  we  nave  more  data  near  the  mode,  and  (b) 

l  2 

we  need  the  values  near  x=l  to  fit  well,  we  have  chosen  x  =  x  and  x  = 

the  midp  it  of  the  next  larger  histogram  interval. 

For  example,  from  the  PFT  dat  i  histogram  at  the  end  of  this 
1  2 

appendix,  we  chose  x=0.9A,  x  =x'“x“0.94,  x  “0.98.  The  results  were:  q+l=342, 
p-l~321.48.  As  mentioned  before,  it  is  critical  to  have  precise  evalua¬ 
tions  of  the  logarithms  in  the  curly  brackets  and  on  the  left.  Here  the 
former  had  the  value  0.0012  and  we  therefore  obtained  q+l=(0.4101)/0.0012 


f, 


Evidently  our  chosen  form  h(x)  can  reasonably  represent  the  non- 
atomic  portion  of  our  observed  sample  histograms  for  PFT/NFT  and  Total  PFT 
and  NFT  Efficiencies.  We  have  thus  achieved  our  end  of  obtaining  a  usable, 


easily  specified,  parametric  class  of  continuous  density  functions  to 
represent  efficiency  populations. 

To  make  comparisons  betwee  1  programs  by,  say,  the  Kullback-Leibler  "diver¬ 
gence"—^  of  these  canonical  density  functions,  we  would  estimate  the 
s,  q  parameters  for  the  NFT  program  just  as  we  have  done  for  the  PFT 
program.  We  do  not  complete  this  numerical  calculation  here,  however. 

Instead  we  simply  display  the  formula  for  determining  the  Kullback-Leibler 
divergence  in  the  following  form. 


1  1  9  (l-fjrl,..  f, 

J(l:2)  =  ;[(l-f.)h1(t)-(l-f.)h2(t)]  Ln  [- - dt  +  (f.-f-Hn  ~ 

o  1  1  (1-f  )hZ(t)  1  1  *2 


where  f ^ ,  f^  represent  the  proportiois  of  efficient  points  for  PFT,  NFT 

respectively — e.g.,  0.35  and  0.38 —  as  in  the  follow  ng  Figures,  and  h^(t) 

-2 

and  h  (t)  are  the  continuous  dpnsity  parts  for  PFT  and  NFT,  respectively, 
as  determined  from  our  canonical  dis  .ribution.  From  this  formula  we  have  vi- 
dently  guaranteed  the  mutual  absolu  e  continuity  properties  that  this 
measure  requires. 


—See  Fullback  [14]  pp  6  ff. 
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